Papers with training stability
Adversarial Text Generation via Sequence Contrast Discrimination (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to generate human-like texts are auto-regressive, but they suffer from exposure bias due to the dependence on the previous sampled output during the inferring phase. |
| Approach: | They propose a sequence contrast loss driven text generation framework which learns the difference between real texts and generated texts and uses that difference. |
| Outcome: | The proposed framework improves training stability and quality of generated texts and avoids the time-consuming sampling process. |
Improving Retrospective Language Agents via Joint Policy Gradient Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have sparked interest in creating autonomous agents. |
| Approach: | They propose a framework that jointly optimizes both task-planning and self-reflective evolution capabilities in language agents. |
| Outcome: | The proposed framework improves task planning and self-reflective evolution capabilities in language agents. |
UNSEE: Unsupervised Non-contrastive Sentence Embeddings (2024.eacl-long)
Copied to clipboard
| Challenge: | Unsupervised Non-Contrastive Sentence Embeddings demonstrates better performance compared to SimCSE in the Massive Text Embing (MTEB) benchmark. |
| Approach: | They introduce UNSEE, which stands for Unsupervised Non-Contrastive Sentence Embeddings, which demonstrates better performance compared to SimCSE in the Massive Text Embing benchmark. |
| Outcome: | The proposed solution achieves better performance than contrastive objectives on the Massive Text Embedding (MTEB) benchmark. |
ScaleBox: Enabling High-Fidelity and Scalable Code Verification for Large Language Models (2026.acl-demo)
Copied to clipboard
Jiasheng Zheng, Xin Zheng, Boxi Cao, Pengbo Wang, Zhengzhao Ma, Qiming Zhu, Jiazhen Jiang, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
| Challenge: | Existing code sandboxes fail to provide accurate verification and efficiency under high-concurrency workloads. |
| Approach: | They propose a high-fidelity code verification system that provides sandbox feedback for RL training and evaluation. |
| Outcome: | The proposed system outperforms heuristic-matching baselines on LiveCodeBench and training stability on high-concurrency workloads. |
Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings (2026.tacl-1)
Copied to clipboard
Miguel Moura Ramos, Tomás Almeida, Daniel Vareta, Filipe Azevedo, Sweta Agrawal, Patrick Fernandes, André F. T. Martins
| Challenge: | Reinforcement learning (RL) is an effective and robust method for training neural machine translation systems. |
| Approach: | They propose a method that leverages fine-grained, token-level quality assessments . they use a state-of-the-art quality estimation system as their token- level reward model . |
| Outcome: | The proposed approach leverages fine-grained, token-level quality assessments along with error severity levels to improve translation quality. |
Bag of Tricks for Sparse Mixture-of-Experts: A Benchmark Across Reasoning, Efficiency, and Safety (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks focus on isolated aspects of MoE, with conflicting conclusions . a lack of consensus on optimal design choices is limiting to specific aspects of the model. |
| Approach: | They propose to evaluate two popular MoE backbones across four dimensions of design choices . they find token-level routing and z-loss regularization improve reasoning performance . |
| Outcome: | The proposed framework evaluates two popular MoE backbones on over eight metrics. |
Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to steer LLMs towards human preference suffer from noisy positive-negative training pairs. |
| Approach: | They propose a distributional preference optimization method which maximizes discrepancy between dispreferred responses and generated non-negative ones. |
| Outcome: | The proposed method achieves comparable generation quality and surpasses the latest strong baselines in producing less harmful and more informative responses with better training stability and faster convergence. |
Interactive Training: Feedback-Driven Neural Network Optimization (2025.emnlp-demos)
Copied to clipboard
| Challenge: | In traditional neural network training, static optimization methods lack flexibility and responsiveness . authors demonstrate that Interactive Training provides superior training stability and reduced sensitivity to initial hyperparameters . |
| Approach: | They propose an open-source framework that enables real-time feedback-driven optimization of neural networks by human experts or automated AI agents. |
| Outcome: | The proposed framework achieves superior training stability, reduced sensitivity to initial hyperparameters, and improved adaptability to evolving user needs. |
BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards (2026.findings-acl)
Copied to clipboard
| Challenge: | Critic-free reinforcement learning with verifiable rewards (RLVR) is a practical paradigm for aligning Large Language Models. |
| Approach: | They propose a framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments. |
| Outcome: | Experiments show that RLVR improves training stability and performance compared to critic-based methods . compared with other approaches, RL VR improves in cold-start regimes with binary verifiers . |
Think Outside the Policy: In-Context Steered Policy Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing Reinforcement Learning from Verifiable Rewards (RLVR) methods exhibit limited exploration due to reliance on on-policy rollouts which are limited to the current policy’s distribution, resulting in narrow trajectory diversity. |
| Approach: | They propose a framework that leverages the in-context learning capability of Large Reasoning Models to provide expert guidance using existing datasets. |
| Outcome: | The proposed framework improves RLVR performance and training stability on mathematical reasoning benchmarks. |
Unsupervised Relation Extraction: A Variational Autoencoder Approach (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for relation extraction use latent variables and supervised training which requires large datasets. |
| Approach: | They propose a VAE-based unsupervised relation extraction technique that uses latent variables as an intermediate variable instead of a latent variable. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the NYT dataset and outperformed existing methods. |
Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RL-based search agents rely on stochastic exploration, leading to inefficient reasoning trajectories and unstable training. |
| Approach: | They propose a framework to enhance the performance and training stability of search agents by transforming raw reasoning trajectories into hierarchical experience knowledge. |
| Outcome: | The proposed framework exhibits strong cross-task and cross-algorithm generalizations on multiple complex agentic search and mathematical reasoning benchmarks. |
Don’t Tell the Answer, Truly Guide the Reasoning During RL Rollouts (2026.findings-acl)
Copied to clipboard
Xinyi Wang, Jinyi Han, Zishang Jiang, Tingyun li, Jiaqing Liang, Sihang Jiang, Zhaoqian Dai, Ma Shuguang, Fei Yu, Yanghua Xiao
| Challenge: | Existing methods such as GRPO often break down when task difficulty exceeds the model’s capacity, resulting in sparse rewards and inefficient training. |
| Approach: | They propose to measure the compatibility between external guidance and a model's intrinsic policy by introducing an adaptive framework to enhance reasoning performance while explicitly preserving high Affinity. |
| Outcome: | The proposed framework outperforms baseline models while maintaining high Affinity. |
CAME: Confidence-guided Adaptive Memory Efficient Optimization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing memory-efficient methods require second-moment estimates of the per-parameter gradients to maintain their performance. |
| Approach: | They propose to use memory-efficient optimizers to reduce memory usage by preserving second-moment estimates of gradients. |
| Outcome: | The proposed method achieves fast convergence and lower memory usage across training tasks. |
Towards Faithful Dialogues via Focus Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge-grounded models rely on elaborate data engineering or increasing the model’s parameters ignoring to track the tokens that significantly influence losses, which is decisive for the optimization direction of the model in each iteration. |
| Approach: | They propose a novel learning approach that adjusts the contribution of each token to the optimization direction by directly scaling the corresponding objective loss. |
| Outcome: | The proposed approach achieves the new state-of-the-art results and generates more reliable responses while maintaining training stability. |
Towards Effective and Efficient Continual Pre-training of Large Language Models (2025.acl-long)
Copied to clipboard
Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Xin Zhao, Zhicheng Dou, Jiaxin Mao, Yankai Lin, Ruihua Song, Jun Xu, Xu Chen, Rui Yan, Zhewei Wei, Di Hu, Wenbing Huang, Ji-Rong Wen
| Challenge: | Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks. |
| Approach: | They propose a Continual pre-training method that can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
| Outcome: | The proposed method can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
SPO: Self Preference Optimization with Self Regularization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing reference-free preference optimization methods exhibit higher training efficiency but are prone to overoptimization, leading to performance degradation. |
| Approach: | They propose a reference-free preference optimization method that replaces the logsigmoid loss function with a SiLU function to improve the model's performance. |
| Outcome: | The proposed method achieves 7% improvement over SimPO on AlpacaEval 2 and MT-Bench. |
Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs. |
| Approach: | They propose a component-wise gradient norm clipping method to adjust convergence speed for different components to improve generalization performance, convergence speed, and training stability. |
| Outcome: | The proposed method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability. |
Scale Down to Speed Up: Dynamic Data Selection for Reinforcement Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to Reinforcement Learning (RL) rely on massive static datasets, leading to computational inefficiency and redundant gradient updates. |
| Approach: | They propose a data-centric RL framework that dynamically selects the most informative training samples to optimize RL for mathematical reasoning. |
| Outcome: | The proposed framework achieves comparable performance to full-data training methods while requiring only 1.5K samples instead of 220K, reducing training time from 13 days to just 4 hours on 8A800 GPUs. |
StablePrompt : Automatic Prompt Tuning using Reinforcement Learning for Large Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have made it difficult to find appropriate prompts for tasks with multiple input-output formats. |
| Approach: | They propose a prompt tuning method based on reinforcement learning (RL) they propose an anchor model and an extension for generating input-dependent prompts. |
| Outcome: | The proposed method outperforms existing methods on a variety of tasks and achieves State-of-the-art performance across diverse types and sizes of LLMs. |
Best Student Forcing: A Simple Training Mechanism in Adversarial Language Generation (2020.lrec-1)
Copied to clipboard
| Challenge: | Language models trained with Maximum Likelihood Estimation (MLE) have been considered as a mainstream solution in Natural Language Generation (NLG) however, they are reportedly suffering from training instability and mode collapse, and therefore outperform conventional MLE models. |
| Approach: | They propose a method to improve Generative Adversarial Nets (GANs) using best student forcing and discriminators to increase training stability and sample diversity. |
| Outcome: | The proposed techniques outperform MLE models and outperformed existing approaches in terms of sample diversity and training stability. |
CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing search agent pipelines rely on sparse outcome rewards, leading to inefficient exploration and unstable training. |
| Approach: | They propose a tool-integrated reasoning framework that provides turn-level feedback via a retrospective critic mechanism. |
| Outcome: | The proposed framework outperforms baselines in multi-hop reasoning benchmarks and achieves faster convergence and training stability. |
Reflective RAG: Self-Evaluation Driven Strategy Optimization in Agentic Retrieval-Augmented Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent agentic RAG systems lack the capacity to evaluate the utility of retrieved information, leading to brittle reasoning and suboptimal decision-making. |
| Approach: | They propose a framework that integrates self-evaluation to dynamically optimize retrieval and generation strategy. |
| Outcome: | The proposed framework outperforms strong agentic baselines on five knowledge-intensive QA benchmarks and improves training stability and generalization to multi-hop reasoning tasks. |
DisLoRA: Task-specific Low-Rank Adaptation via Orthogonal Basis from Singular Value Decomposition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments on GLUE and Commonsense Reasoning benchmarks demonstrate that DisLoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, and SORSA. |
| Approach: | They propose a framework that leverages singular value decomposition to decompose pretrained weight matrices into orthogonal backbone and task-specific subspaces. |
| Outcome: | Extensive experiments on GLUE and Commonsense Reasoning benchmarks show that DisLoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, and SORSA. |
BranchNorm: Robustly Scaling Extremely Deep Transformers (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work on DeepNorm scales Transformers into extremely deep (1000 layers) due to the training instability of Transformers, the depths of these SOTA models are still relatively shallow. |
| Approach: | They propose a branch-rescaled model which dynamically rescales the non-residual branch of Transformer in accordance with the training period. |
| Outcome: | The proposed approach significantly outperforms existing shallow models on multiple translation tasks and achieves better training stability and convergent performance. |
IDC: Boost Text-to-image Retrieval via Indirect and Direct Connections (2024.lrec-main)
Copied to clipboard
| Challenge: | Dual Encoders (DE) and Cross Attention (CA) frameworks for image and text retrieval are more accurate but slower. |
| Approach: | They propose a dual encoders-based approach to map image and text inputs into a coordinated representation space and calculate their similarity directly. |
| Outcome: | Extensive experiments on the MSCOCO and Flickr30K datasets validate the effectiveness of the proposed methods. |
Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RLVR algorithms suffer from entropy collapse, leading to premature determinism and unstable optimization. |
| Approach: | They propose an adaptive entropy flow balancing mechanism that rescales entropic-increasing and enotro-decreazing updates according to their contributions to enthroy change. |
| Outcome: | The proposed method outperforms existing RLVR algorithms on six reasoning benchmarks. |
HyperLoRA: Efficient Cross-task Generalization via Constrained Low-Rank Adapters Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to adapt pre-trained language models (PLMs) to emerging tasks are costly and inefficient. |
| Approach: | They propose a meta-network that generates task-specific weights without any optimization. |
| Outcome: | The proposed approach has flexible generalization ability and superior performance over hypenetworks. |
Graph-GRPO: Stabilizing Multi-Agent Topology Learning via Group Relative Policy Optimization (2026.findings-acl)
Copied to clipboard
Yueyang Cang, Xiaoteng Zhang, Erlu Zhao, Zehua Ji, Yuhang Liu, Yuchen He, Zhiyuan Ning, Chen Yijun, Wenge Que, Li Shi
| Challenge: | Recent approaches to optimize communication topology rely on single-sample policy gradients with absolute rewards. |
| Approach: | They propose a topology optimization framework that integrates Group Relative Policy Optimization. |
| Outcome: | The proposed topology optimization framework outperforms state-of-the-art methods on reasoning and code generation benchmarks. |
Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Current reranking models are optimized on static human annotations in isolation, decoupled from the downstream generation process. |
| Approach: | They propose a reinforcement learning framework that directly aligns reranking with LLM's generation quality. |
| Outcome: | Experiments on knowledge-intensive benchmarks show that RRPO outperforms strong baselines. |
SAFO: Stable Adaptive Fairness Optimization for LLM-Based Social Survey Simulation (2026.acl-long)
Copied to clipboard
| Challenge: | Social survey simulations are increasingly used to improve minority performance and social-welfare metrics. |
| Approach: | They propose a dynamic utility–fairness optimization framework for LLM-based survey simulation that explicitly targets fairness and training stability. |
| Outcome: | The proposed framework improves minority performance and social-welfare metrics on three large-scale survey datasets from China, the U.S. and Europe. |
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via sequence-level likelihood (2026.acl-long)
Copied to clipboard
| Challenge: | Group Relative Policy Optimization (GRPO) has significantly advanced the reasoning ability of large language models (LLMs). |
| Approach: | They propose a token-level framework that leverages sequence-level likelihood to link group-level rewards with individual tokens via token- level aggregation and introduces a KL-Divergence mask constraint that targets tokens with positive advantages and decreasing entropy to mitigate abrupt policy updates. |
| Outcome: | Experiments show that TEPO achieves state-of-the-art performance on mathematical reasoning benchmarks and reduces convergence time by 50% compared with GRPO/DAPO. |
STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing GRPO-based methods allocate sampling uniformly across tasks regardless of difficulty, propagate misleading learning signals and incur high sample-collection costs. |
| Approach: | They propose a framework that allocates sampling based on per-task success rates and performs fine-grained step-level optimization. |
| Outcome: | The proposed method improves sample efficiency and training stability over existing GRPO variants and three ablation variants on OSWorld and AndroidWorld. |
How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients (2026.acl-long)
Copied to clipboard
| Challenge: | Spectral properties of low/high-quality instruction and reasoning data are used to explain finetuning dynamics in large language models. |
| Approach: | They propose to analyze layer-wise gradients induced by low/high-quality instruction and reasoning data for LLM post-training. |
| Outcome: | The results show that higher-quality data are associated with lower nuclear norms and higher effective ranks. |
Orthogonal Finetuning Made Scalable (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent shift in foundation models has slowed the adoption of finetuning methods . however, its high runtime and memory demands limit its scalability . |
| Approach: | They propose an input-centric reformulation that uses matrix-vector multiplications instead of cubic multiplication . they extend OFTv2 to support finetuning quantized foundation models and show it outperforms QLoRA . |
| Outcome: | The proposed model outperforms the popular QLoRA in training stability, efficiency, and memory usage. |
LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for implementing large language models are limited by high computational and memory requirements. |
| Approach: | They propose a lightweight binarization framework that achieves effective W(1+1)A4 quantization through a novel three-stage quantization strategy. |
| Outcome: | The proposed framework surpasses state-of-the-art methods on W2A4 quantization settings across languages. |
How to Set the Learning Rate for Large-Scale Pre-training? (2026.findings-acl)
Copied to clipboard
| Challenge: | Optimal configuration of the learning rate (LR) is a fundamental yet formidable challenge in large-scale pre-training. |
| Approach: | They propose a Fitting Paradigm and a Transfer Paradigme to investigate fit and transfer . they propose scalability and elucidate the reasons why module-wise parameter tuning underperforms . |
| Outcome: | The proposed model reduces the search complexity by reducing the search cost by lowering the search factor. |
Entropy-Aware Reshaping of Reinforcement Signals for Multi-Answer Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards (RLVR) is a standard post-training paradigm for large language models. |
| Approach: | They propose a framework that reshapes how learning signals are normalized and aggregated. |
| Outcome: | Experiments on MCTACO and MMLU-Multi show that the proposed framework improves accuracy, training stability and cross-dataset transfer performance. |
Understanding the Behaviors of Environment-aware Information Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Recent retrieval-augmented generation approaches have demonstrated strong capability in handling complex queries. |
| Approach: | They propose a branching-based rollout technique that improves training stability . they find different retrievers exhibit distinct optimal query styles . |
| Outcome: | The proposed method improves training stability and improves retrieval-aware systems. |
TRAC: Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Current reward models for reinforcement learning (RL) rely on outcome rewards that propagate a single scalar value across all tokens based on final correctness. |
| Approach: | They propose a framework that derives dense token-level supervision from LLMs . they use a multi-granularity calibration mechanism to modulate teacher influence . |
| Outcome: | The proposed framework evaluates teacher reliability across problem-level expertise, trajectory-level discrimination, and token-level confidence. |